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(57) Abstract 

In a method of estimating motion vectors, motion param- 
eters (pl-p4) are dctcnmincd (PC) for a given ficW (n-l) of a 
video signal; and motion vectors (D(X, n)) for a field (n) of 
said video signal arc dcicnnincd in dependence upon at least 
one predctcmiincd motion vector (D(x-X. y-Y, n), D(k+X, y-Y. 

y+2Y, n-I)) and at least one additional motion vector 
(PV(X)) derived (PG) from said moUon parameters (pl-p4). 
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. Motion estunation 



The invention relates to a method and device for estimating motion vectors, and 
to a video display a^jparatus comprising such a device. 

EP-B-0,410,826 [1] discloses a method of performing a recursive motion 
5 estimation between a reference image and a current image, to realize a fixed number of 
iterations, each iteration consisting of; 

. sampling both images with a resolution which increases at each iteration; 

- detecting characteristic samples in the reference image; 

- estimating a local motion vector for each characteristic sample in the reference image, in 
10 which the estimation is iniriaHTj^fi with a displacement vector derived from a global motion 

estimation realized in the course of the iteration preceding the current iteration; and 

- carrying out a global motion estimation of the current image, in which values of parameters 
of a model are nl^Mlat'v^ on the basis of the local motion vectors estimation in the course of 
the latest iteration. 

15 To estimate the global motion, the values of 8 parameters al a8 defining a 

motion model are calculated, in such a manner that a sample having coordinates X, Y is 
displaced to coordinates X', Y*, where 
X* = ( al.X + a2.Y + a3 ) / ( a7.X + aS.Y + 1 ) and 
Y' = ( a4.X + a5,Y + a6 ) / ( a7.X + aS.Y + 1 ), 

20 the 8 parameters al a8 being calculated by solving a system of equations with 8 unknowns 
which is constituted by the two relationships above, written for characteristic samples, the 
coordinates X', Y* being determined as a function of an estimated local motion vector for 
each of these samples and as a function of the coordinates X, Y of the sample under 
consideration. ^ 

25 To estimate a local motion vector for each characteristic sample of the current 

image, the following steps are carried out; 

- defining a first block of samples of the reference image, centered on the characteristic 
sample under consideration; 

- defming a plurality of second blocks of samples of the current image, centered on samples 
30 conesponding to the characteristic sample under consideration but shifted over a plurality of 
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vectors to be tested, each vector to be tested being constituted by the vector sum of an initial 
di^laccment of the charactcrisdc sample under consideration cadculatcd from the ^global 
estimate of motion made in the course of the iteration preceding the iteration in progress, and 
from an additional displacement taken from a plurality of predetermined additional 
5 displacements; and 

- selecting the vector to be tested which yields the best match between the respective second 
block and the first block to obtain the local motion vector. 

It thus dppcm that in the prior art, a plurality of iterations is carried out 
between the same images to obtain the final motion vectors corresponding to the highest 
10 resolution. 



It is, inter alia, an object of the invention to provide a less complex motion 
vector estimation. To this end, a first aspect of the invention provides a method of estimating 
15 motion vectors as defined in claim 1. A second aspect of the invention provides a motion 
vector estimation device as defined in claim 10. A third aspect of the invention provides a 
video display ^jparanis as defined in claim 11. Advantageous embodiments are defined in the 
dependent claims. 

In a method of estimating motion vectors in accordance with a primary aspect 
20 of the present invention, motion parameters arc determined for a given field (n-1) of a video 
signal; and motion vectors for a field (n) of said video signal arc determined in dependence 
upon at least one predetermined motion vector and at least one additional motion vector 
derived from said motion parameters. 

These and other aspects of the invention will be apparent from and elucidated 
25 with reference to the embodiments described hereinafter. 



In the drawings: 

Fig. 1 shows linearly changing motion velocities due to zooming; 
30 Fig. 2 shows nine vectors taken from the temporal prediction memory of a 

motion vector estimator, 

Fig. 3 shows the mean squared errors (MSE) for the 3D-RS block matching 
motion vector estimation with and without the new parametric vector candidate; 

Fig. 4 shows vector x-componenu without parametric candidate (upper half of 
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* Fig. 4), and with parametric candidate (lower half of Fig. 4); 

iFig. 5 shows vector y-components without paran^tric candidate (upper half of 
Fig. 5), and with parametric candidate Oower half of Fig. 5); 

Fig. 6 shows an embodiment of a di^iay apparatus comprising a motion vector 
5 estimation device in accordance with the present invention; and 

Fig. 7 illustrates in a schematic form various embodiments of an image display 
apparatus having a motion vector furnishing device in accordance with the present invention. 



I. Introduction 

10 Motion esdmators minimize or maximize a, sometimes complex, but often 

simple, criterion function, selecting one of various possible motion vectors for every location 
in the image. They either apply the exhaustive search method, i.e. try all possible vectors in 
a predefined range, to obtain the global optimum of the criterion function, or use one of the 
efficient ^proaches and test a linuted number of candidate vectors only [9,10,n»13]. In the 

15 more advanced efficient motion estimation methods this limited candidate set contains the 
most likdy, prediction, vectors. The likelihood can be based on analysis of the picture at a 
higher hierarchical level [2,14], by analysis in the frequency domain [15,16,17], or the 
likelihood is based on spatial and/or temporal proximity of the vectors in the candidate set 
[4,5,6.12,19]. 

20 Motion in video images is either due to object motion, or caused by camera 

movements. Object size causes correlation of motion vectors in the spatial domain, while 
object inertia explains temporal correlation of vector fields. The category of camera motion 
includes motion due to pans, tilts, and travels of the camera, and zooming with its lens. This 
type of motion usually causes very smooth velocities in the spatial and in tiic temporal 

25 domain. A zoom with the camera lens results in motion vecton that arc lineariy changing 
with the spatial position as illustrated in Fig. 1 in which Vv indicates linearly changing 
vertical velocities and Vh indicates lineariy changing horizontal velocities. A pan, tilt or 
travel with the camera, on the other hand, causes a uniform motion vector for the entire 
television screen. These types of motion can be described with a three parameter model, as 

30 has been suggested before [3]. Also methods have been suggested in the literature how to 
extract the parameters of such a model from a rough motion vector field [1]. Even more 
complex global motion, Uke roUtion, can be described with a parametric model, which than 
needs up to eight parameters [18]. In this description, we propose to use tiie rough estimate 
of the motion vector field in the previous image for generating, with a simple parametric 
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description of this field, an additional prediction vector for every location in the picture, "ptis 
prediction is included in the candidate set of an advanced efficient block-matcher, that further 
includes temporal and spatial prediction vectors and an update thereof [4,6]. As the 
parametric model enables calculation of vectors even at the boundaries of the image and, if 
5 required, with sub-pixel accuracy, it can bring a considerable improvement in case of camera 
motion, particulariy if there are areas with litflc or low-contrast detail near the boundaries of 
the image, where temporal and spatial predictors sometimes fail. 

Section n of this description introduces our notation and summarizes the motion 
10 estimation algorithm of [6], which we used as a basis to validate our improvement. 
Section m introduces the parametric model and the extracticm of the additional candidates 
from previously calculated motion vector fields. Section IV gives an evaluation of the 
improvement in a critical plication (de-interlacing), section V describes how the 
s^licability of MPEG motion vectors can be extended by using the principles of the present 
15 invention, and section VI summarizes our conclusions. 



n. The 3-D Recursive Search Blo clc-Matcher 

As we considered perfection of an advanced motion estimator most interesting, 
we applied the high quality method of [6]. This algorithm yields a quarter pixel accuracy and 
20 a close to true-motion vector field, relevant for scan rate conversion [4,5]. Furthermore, the 
efficiency of this algorithm is such that it is currentiy the only single chip true-motion 
estimator [7,8]. 

In block-matching motion estimation algorithms, a di^laccment vector is 
assigned to the center X of a block of pixels BQQ in the current field n by searching a similar 
25 block within a search area SAQO. also centered at X but in the previous field n-l. The 
similar block has a center which is shifted with respect to K over the displacement vector 
n). To find U(K, n), a number of candidate vectors C are evaluated applying an error 
measure e(Q, X to quantify block similarity. 

More formally, C5~" is defined as the set of candidate vectors £, describing aU 
30 possible (usually integer) displacements with respect to K within the search area SAQO in the 
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• previous image: 

where and M axe constants Umiting SA(X). Furthermore, a block B(K) centered at X and of 
size X fry y consisting of pixel positions i in the present field n, is now considered: 

The displacement vector QGL n) resulting from the block-matching process, is a 
5 candidate vector C which yields the minimum value of an error function 6(0, X. n): 

mX. n)€(C€ CS- 1 .(C X. n) s .(£. X. n) V£ 6 CJ- ) 

If, which is the common case, the vector n) with the smaUest matching 
error is assigned to all pixel positions x >n the block BQD: 

VieBCX): (4) 
iKi «) e (C e «-« I .(C «) ^ •(£ i ») V£ e CS— } 

rather than to the center pixel only, a large reduction of compuUtions is achieved. As an 
implication, consecutive blocks B(X) are not overlapping. 

10 

The eiTor value for a given candidate vector £ is a function of the luminance 
values of the pixels in the current block and those of the shifted block from a previous field, 
summed over the block B(X). A common choice, which we will use, is the Sum of the 
Absolute Differences (SAD): 



»(C i n) = SAD - J) IFKx, n)-F(x-C «) I 



(5) 



15 

Alternatively, a Mean Square Error (MSB) or a Normalized Cross Correlation 

Function (NCCF) may be used. 

Rather than calculating all possible candidate vectors, the 3-D Recursive Search 
block-matcher of [6], takes spatial and/or temporal "prediction vectors" from a 3-D 
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neighborhood, . and a single updated prediction vector. This implicitly assumes spatial and/or 
temporal consistency. The updating process involves updates added to either of the! spatial 
predictions. We applied a candidate set CSQL n) containing candidate vectors Q, from which 
the block-matcher selects its result vector.^, delined by: 



C5(X, n) = 



(6) 



5 where the two options for candidate vector £, alternate on block basis, and the update vector 
IZlX n) is taken from a limited fixed integer update vector set US», in our case: 

-.■'(S).(?).W.(5).(°J.(i).(o'M2Mo')' 

To realize sub-pixel accuracy, the update set of equation (7) is extended with 
fractional update values. We realized a quarter pixel resolution by adding the following 
fractional vectors to the update set: 

10 Because of the small number of candidate vectors, the method is very efficient. 

Furthermore, due to the inherent smoothness constraint, it yields very coherent vector fields 
that closely correspond to the true-motion of objects. This particularly, makes it suitable for 
scanning format conversion. 



15 m. Camera motion and the parame tric model 

Camera motion includes motion due to panning, tilting, riding, and zooming of 
the camera. This type of motion has a very regular character causing very smooth velocities 
i.e. motion vectors, compared to object motion. Zooming with the camera will generate 
motion vectors that are linearly changing with the spatial position, as illustrated in Fig. 1. 

20 Panning, tilting or riding with a camera, on the other hand, will generate a uniform motion 
vector for the entire television screen. 
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m.l A three p arameter model 

These types of motion can be described with a three parameter model, as has 

been suggested before [3]: 



in which pi describes the panning, p^ the tilling, and py the zooming of the camera. This 
model is valid only if the blocks arc exactly square on the image plane. At least in television 
this assumption is often invalid, even when the blocks are square on the pixel grid, due to 
the use of various aspect ratios with fixed sampling frequency. 

In these cases a four parameter model can be used according to: 

^ , . (10) 



(Pxin) ^Py(n)x 

"[p2('^) ^P4(^)y, 



where a fixed ratio results between and p*, which is determined by the ratio of the 
10 horizontal (H) and vertical (V) sampling density. If the ratio pyfp4 does not correspond to the 

ratio of the H and V sampling density, the extracted parameters must be unreliable. It is 

possible therefore, by measuring this ratio, to identify situations in which the additional 

candidate is better not used in the motion esdmator. 

Extending the model to a six parameter model renders it furthermore possible to 
15 include rotations. This type of motion, however, is not very likely in television broadcast 

material, and therefore not further considered in this description (although not excluded from 

the claims). 

TTT l. TTpfodinP the ^-D RS Block-Matcher with a Parametric candidstg 
20 An implementation of the proposal results by adding a single candidate vector 

for every block to the set CSPC, n) defined in section n. where this additional candidate 
QOL n) for the block BQD is calculated according to the four parameter model of equation 
(10), i.e. it results: 

'•>"(p,(«).p,(n)yj 

Rather than adding a candidate vector to the motion estimator it is possible to 
25 alternate (e.g. on block base) this additional candidate with another candidate vector. In this 
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case the operations count of the estimator hardly increases, while the advantage was found to 
be comparable. As a consequence candidate vector Q in equation -(6) is modiTied to: . 



C^(X. n) 



C(X + (2y)' , even blocks 



A refinement of this thought is to have the occurrence frequency of the 
parametric model candidate dqxnd on the reliabiUty of the model. In our implementation this 
5 led to a further modification of C according to: 



p,(n) *p,(n)X\ (oddblocks A model reliable) 



In the next subsection we wiU describe a parametric model extraction procedure 
that allows such a reliability indication. 

F-Ttraction of tb ft parameters ffntn the image data 
10 There are many options to extract the parameters of a global motion model 

from an estimated motion vector field. In our case where the model is integrated in the 3-D 
RS Block-Matcher, it makes sense to start from already available motion vectors, i.e. the 
vectors available in the temporal prediction memory. To keep the operations count low, it is 
furthennore attractive to use a limited set of the vectors available in this memory only. 
15 As a consequence of the choice to use vectors from the prediction memory, the 

vaUdity of the candidates gcneiated with the parametric model increases in time due to 
recursivity of this approach. If the parametric model is vaUd for significant portions of the 
image, the additional candidate will be selected by the estimator engine more often, which in 
the next field improves the quality of the parameters extracted from the vectors in the 
20 temporal prediction memory. This again improves the accuracy of the generated additional 
candidate, it wiU be selected more often, and the model wUl improve further, etc.. 

To estimate the parameters of the model describing the global moUon, we took 
a sample set S(n) containing 9 motion vectors. HQL n-J) from different positions X on the 
block grid in a centered window of size {W-2m)X x (H-2q)7 in the picture with width WX 
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• ' ■ 

, and height HY from the temporal vector prediction memory according to: 
Sin) » (IKL /I - 1) 1 X = -(»72 - m)X, 0, +( »72 - Jf. 

(14) 

Xy = -(H/2-g)r. 0. +(///2-g)r} 

where m and 9 are small and depend on the number of blocks in the image. Fig. 2 illustrates 
the position in the image from which the vectors in the sample set are taken. Choosing m and 
^ to be small, generally yields a more accurate estimate of the zoom parameters and p^. 
5 Vectors too close to the image boundary, however, can be unreliable in case of fest camera 
motion. It will be clear that other choices are possible that might prove to be better in 
specific cases, but we see no evident cause for systematical improvement with another set of 
similar size. Larger sample sets can improve the result, but also increase the calculational 
burden. 

10 A four paiameter model can be solved with four equations, i.e. two 

"independent" sample vecton. Indq)endent, here, means that these vectors are not taken 
from the same horizontal or vertical position of the image. The number of combinations to 
choose r samples from a set of 5 is known from elementary statistics: 

{r\ s\ (15) 



W (J - r)\r\ 



. (M€N^) 



In our case we arrive at j=9 and r =2, which implies 36-18 = 18 options to solve the model, 
15 as there are 36 combinations of 2 out of 9, and 18 pairs of "dependent" samples, i.e. 
samples on the same row or column. There are three rows and three columns, each enabling 
three pairs of two vectors bdng selected from the available 3 vectors, so 2x3x3=18 
dependent pairs. 

Extraction of the parametcn from the i* pair of independent sample vectors 
20 C(2C/. n-i) and RiXn* bo* in S(n), is straightforward: 

P^W = y _Y 



and: 



SUBSTITUTE SHEET (RULE 25) 
Printed from Mimosa page -11- 



wo 97/46022 PCT/IB97/00548 

10 ... 



while: 
and finally: 



(18) 



(19) 



Now for every pair of two independent sample vectors a set of four parameters 
is available, and the best parameter set has to be selected. We propose here to assign the 
5 median value of all options for each parameter to the eventual model parameter. 
Consequently, for i=l up to i=18 we rank the individual parameter values such that: 



ViT2i/il7Avyil s>54: Pj'\n) ^pjin) ^Fj'\n) 



(20) 



and derive the final value of the model parameters according to: 
Vyi 1 i y i 4: p/n) = \{p^{n) * (^/"(n) ) (21) 



Finally, the ratio of pa and is calculated to check the reliability of the model. 
In the experiments we allowed 25% deviation from the nominal value. If the parameters pass 
10 this test and the ratio should be pjp^ = R according to the H and V sampling densities (see 
section III.1), it is possible to slightly further improve the accuracy of the model by 
rqjlacing p^ and p^ by their average value after correcting for the different sampling densities 
( becomes: (py ^ R, p^^ll, and p^ becomes: (p* + i/R- P3)/2 ). This sophistication was 
implemented in the evaluated algorithm. 

15 

In an alternative embodiment which is favorable as it requires less computations 
than the median algorithm of equations (20) and (21). the set of four parameters for every 
pair of two independent sample vectors arc used in the following manner. The 18 different 
sets of parameters arc cyclically used (one pair for each field) until the model becomes 
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reliable, i.c until 'a correct p3/p4 ratio is obtained. The set which yielded the correct p3/p4 
' ratio is maintained until it appears that the p3/p4 ratio is no longer correct. In that case, the 
parameter sets are again cyclically applied until the correct p3/p4 ratio is obtained again. 

5 There arc many options to extract the parameters of a global motion model 

from an estimated motion vector field. Furthermore, there are many possible motion 
estimation algorithms in which an additional candidate vector generated with such a 
parametric motion model could be applied. Another possible implementation of the invention 
in the 3-D RS block-matching motion estimator of [6] will be described below. 

10 To estimate the parameters of the global motion model, nine vectors, DI0..8] 

assigned to different positions in the picture are taken from the temporal vector prediction 
memory as illustrated in Fig. 2. Four vector values will be selected from this vector set to 
calculate the parameters. In order to select the most reliable base for the parametric model 
(the best four vectors out of the nine available ones), first three average x-components DjJqJ 

IS are calculated according to: 

fi^m-(fl/ii>i2;2])/2 

i2^ra-(fl[3]^I2;4])/2 (22) 
C^[3]-(i2^5]*C[61)/2 

To each of these averages a quality measure Diffjq] is assigned using: 
Diffjil]^absilljl]-Iip]) 

Diffjl] = absiUJl] ' HJiO] ) 03) 

The background of this quality naeasure is that the calculated differences should 
be zero if the vectors are only due to global motion (pan or zooming of the camera). The 
larger a certain differeiure is, the larger the chance that the corresponding vectors arc a poor 
20 base for the parametric model. Similar calculations are required for the vector y-component: 

J2^^P] = (Z2^[6]^C[41)/2 (24) 
fi^,[3]=(i2;5]-Ct31)/2 

and: 
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DWp]-absiIl^p]-]2J[a\) • (25) 

The vector x or y component values belonging to the smallest difference are 
used to interpolate/extrapolate the new parametric subpixcl vector candidate using: 

X, - , ( « =1 V 3 V 5 , AV^: Diffjim] i Dijfjiq] ) 

^f>l^^#J^]) (26) 




rf-f7iRTV<-V7; AVg: D^^myi. Diff^q] ) 
y,-DyIml, (»=3V4V8, AVg: DOfylw] ) 

The gradient P3 of the model must be calculated, as well as the parameters p, 
and P2 indicating the center of the zoom for respectively the vector x_component and the 

5 vector y_component. The gradient pj, is the same for the x_ and y_component, due to 
panning and zooming properties. The gradients are calculated separately. When the separate 
gradients are significantly different, the parametric vector candidates will be regarded 
unreliable and this information can be used to nKxiify the occurrence frequency of the 
parametric candidate as mentioned above. To avoid small disturbances, both gradients are 

10 averaged, giving the final gradient pj. 

where and Ny are the horizontal and vertical distance measured on the block grid between 
the posidofls of the vectors on which the gradient calculation was based. The parameters p, 
and p2 can be found by extrapolation, making sure that the interpolation starts at the edges of 
the vector field: 

(28) 

15 where X[5] and Y15] are the coordinates of D[51 on the block grid. The parametric vector 
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candidate (one per block) can then be calculated as: 

which can than be added to the candidate vector set of any motion estimation algorithm. 

It is good to notice that the reliability criterion mentioned above has to be 
5 adapted in case of geometrical distortions in the processing between camera and motion 
estimator. These distortions (e.g. horizontal compression or vertical stretching) arc 
sometimes introduced by circuits for wide-screen television. 

TV. Evaluation of the im provement 

10 The expected advantage of our proposal was an increased accuracy of the 

estimated motion vectors in case of camera motion. As the original 3-D RS block-matching 
algorithm is already very accurate, a critical application is required to show the 
improvemcait. We found in an earlier stage already [6], that de-interlacing requires a high 
accuracy from the motion estimator, i.e. the sub-pixel fraction of the estimate plays an 

15 important role in the de-interiacing quality. Therefore, here again, we appUed the 3-D RS 
block-matcher in the time-recursive de-interlacing algorithm of Wang et al. [20], to verify 
the advantage of tiie parametric candidate vector. We essentially calculate a motion 
compensated mean square prediction error, similar to what we used in [6], to get an 
indication of the quality of the motion vectors. 

20 More precisely, the MSE is calculated in the measurement window MW of 

(W-2m)X by (H-2q)Y pixels, indicated in Fig. 2, between the luminance of the current input 
field F(s. n) and the previously calculated sequentially scanned picwre F,(i, n-1) shifted over 
the estimated displacement vector D(z, n): 

''^«"> ■ (.K-^)X.(»-^>r .L<^" -'^-'^ " 

where x runs through aU positions in the naeasurement window on the pixel grid of the odd 
25 lines in an odd input field, and through all pixel positions of the even lines in case of an even 
input field. 

In Fig, 3 we show the value of MSE(n) (vertical axis) for a number of fields 
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(horizontal axis) with (curve b) and without (curve a) the parametric caivdidate vectors in the 
estimator of [6J. It can be clearly seen that both algorithms converge in the first fields, and 
that the parametric candidate gives a significant improvement of the performance in the later 
fields. The pictures were selected to provide critical test material. The tested sequence 
5 contains large areas with much vertical and horizontal detail, and movement in various 
directions with a large number of (sub-pixel) velocities due to zooming of the camera. It 
appeared that the additional candidate does not degrade the performance in case no camera 
motion is present. We found that the MSE score on zooming or panning picture material, 
was some 12% lower with the new parametric candidate. 

10 

Figs. 4, 5 visualise the results of the proposal. In these figures the simulation 
results are shown, indicating the velocities with (lower half of Figs. 4, 5) and without (upper 
half of Figs. 4, 5) the parametric vector candidate. The vector x-component is shown in 
Fig. 4, whereas the vector y-componcnt is shown in Fig. 5. The different shades of grey 1-6 
15 in Fig. 4 and 1-5 in Fig. 5 indicate mutually different motion vector component values. In 
this case the input for the model was a camera zoom which, as shown in Fig. 1, should 
result in linearly changing velocities in both the horizontal and the vertical directions. From a 
comparison between the upper and lower halves of Figs. 4. 5, it foUows that due to the 
measure of the present invention, the moticm estimator accuracy has been greatiy improved. 

20 

Fig. 6 shows an embodiment of a display apparatus comprising a motion vector 
estimation device in accordance with the present invention. A television signal received by an 
antenna A is ^lied to a processing circuit P for tuning, demodulation and video signal 
processing operations well known to those skilled in the art An output of the processing 

25 circuit P is applied to a first field memory FMl for obtaining a video signal at a doubled 
field frequency. An output of the first field memory FMl is ^plied to a second field 
memory FM2 to obtain a field-delayed video signal. An input and an output of the second 
field memory FM2 are applied to a motion vector estimation arrangement ME to obtain 
motion vectors n(S, n). The motion vector Q(x, n) and the input and the output of the 

30 second field memory FM2 arc applied to a motion compensated interpolator I, an output of 
which is applied to a display unit Dp. 

The motion vector estimator ME comprises an update generator UG including a 
mod p counter to which a block frequency clock signal Nm is applied. An output of the 
counter C is applied to a look-up table LUT to obtain update vectors II(X. n) as described 
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above, and an 'updatc-iependent penalty TQJ) which increases with an increasing update 
vccjor lengtiti. The update vectors n) are applied to an update and multiplex circuit UP- 
MUX which also receives a zero vector Q, a paramettic candidate vector EVQD from a 
parametric candidate generator PG, and previously determined vectors Ii(x+X, y-Y, n) and 

5 Ii(x-X, y-Y, n) from a prediction memory PM. Tlie update and multiplex circuit UP-MUX 
applies three candidate vectors to a best matching vector selector SEL which also receives a 
vector y+2Y, n-1) from the prediction memory PM. The best matching vector selector 
SEL further receives the input and the output of the second field memory FM2, and 
determines the best vector I2(X> n) between a block in a current field axui a corresponding 

10 group of pixels in a previous field. The best vector OCX, n) is applied to the prediction 
memory PM, and to a block erosion circuit BE to obtain a motion vector n(X. n) for each 
pixel X in the block X. From the prediction memory PM, the sample vectors shown in Fig. 2 
are applied to a parameter calculator PC to obtain four motion parameters which are j^plied 
to the parametric candidate generator PG. 

15 

V. Extending the applicability of MPEG motion vectors. 

DE- A- 195. 10,389 [21] discloses a method and a circuit arrangement for 
receiving coded video signals. The method concerns a video signal which contains a 
sequence of images with compressed redundancy, e.g. coded in accordance with the MPEG2- 

20 standard. The motion decoding is carried out first. The post-processing of the motion- 
decoded fields, which post-processing essentially contains inteipolaticms, is carried out next 
together with the processing to remove flicker. Advantageously, the motion vectors retrieved 
at the motion decoding, are used also for the flicker removal. 

As mentioned above, motion estimation can be seen as an optimization problem: 

25 a sometimes complex, but usually simple criterion function has to be minimized or 
maximized to find an output motion vector. There are brute force methods that simply try all 
possible vectors, in a predefined range, in order to be sure to obtain the global optimum of 
the criterion function. Also there arc efficient approaches that lest only the most likely 
motion vecion. This likelihood is usually determined by spatial or temporal proximity, and 

30 consequently temporal and spatial prediction vectors have been popular in the efficient 
motion esdnxation algorithms. Depending on the motion estinudon algorithm used, the 
properties of the resulting motion vector fields are different, so that the application areas of 
the various motion estimation algoritiims are different. Motion vectors can be used in high 
quality video signal processings like motion compensated scan conversion, motion 
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compensated noise reduction and inuige coding. Tlie intended processing puts constrainte on 
various quaUty aspects of the motion vectors. For example; high qaaUty motidn compensated 
scan conversion requires a motion vector field where the vectors represent the true motion 
vecton (i.e., true direction and speed) within the image sequence. For coding appUcations 
5 this is irrelevant, as the only quality criterion is a low average prediction cnor. 

Brute force methods like full search block matching tend not to give true motion 
vectors. Instead, fuU search block matching minimizes the error between blocks, so its 
vectors will connect two blocks that have a minimal enor between these two blocks, which is 
suitable for minimizing the prediction error in coding ^pUcations. WhUe these minimal 
10 erron work efficientiy for compression algorithms like MPEG, the resulting vectors are 
unsuitable for use in high quality scan conversion. On the other hand, more efficient motion 
estimators like 3-D recursive block matching tend to give true motion vectors that can be 
used in high quality scan conversion but which may not be suitable for use in compression 
algorithms. 

15 With the emergence of MPEG and other types of video compression it is 

possible to have input video material that already contains motion vectors, so at first sight it 
may seem ttiat for this type of source material motion estimation at the receiver side is not 
necessary. However, it is not certain that these motion vectors can be directly used for e.g. 
scan rate conversion, because it is unknown whether the MPEG motion vectors represent 
20 true motion or not. At the decoder/receiver side it is unknown what type of motion 
estimation algorithm was used at the encoder/transmitter side, so one must assume, as a 
worst case situation, that the MPEG motion vectors are optimized for an efficient 
compression and tiiat they do not represent true motion vectors. For example, periodical 
structures and noise in picture areas with UtUe detaU. may cause such inconsistent vectors. 
25 Another problem with motion vectors in MPEG data streams is that it is uncertain that aU 
motion vectors are transmitted within the data stream. For these reasons, the technique 
proposed by [21] seems not to be suitable for high quality post-processing. 

In an attempt to solve a similar problem, it has been proposed (221 to smooth 
the vectors transmitted with a digitally encoded television signal before using these vectors 
30 for a field rate upconversion operation. 

It is an object of a further aspect of the invention to furnish motion vectors 
which are suitable for high quality post-processing. This further aspect of the present 
invention is based on the recognition that the problems described above do not exclude the 
MPEG motion vectors entirely from use in high quaUty scan conversion. When an 
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appropriate post-processing is applied, the MPEG motion vectors can be made useful. More 
specifically, when the receiver is able to determine the quality of the MPEG motion vectors, 
and when it is able to improve the quality of the MPEG motion vectors so that they meet 
certain criteria for the intended processing, the MPEG motion vectors can be used. Several 
5 possible post processing methods to extend the applicability of the MPEG motion vectors are 
considered: 

1. Process the MPEG motion vectors with an operation like some sort of vector 
smoothing operation to increase the usability of the MPEG motion vectors. 

2. Use the MPEG motion vectors as additional candidates along with spatial and/or 
10 temporal candidates in, for instance, a 3-D recursive block matching algorithm, similar to the 

use of an additional parametric vector candidate as disclosed in the non-prepublished EP 
priority patent application no. 96201462.7 fded on 24.05,96 (attorneys' docket PHN 15,824). 

3. The MPEG motion vectors can be used to calculate the parametric vector 
candidates described in the non-prepublished EP patent application no. 96201462.7. 

15 For a schematic description of the various embodiments, reference is made to 

Fig. 7. 

Fig. 7 shows a basic section B, and four sections El E4 showing four distinct 
ways to augment the basic section B to obtain four distinct embodiments B+El, B+E2, 

20 B+E3, and B+E3+E4 of an image display apparatus. 

In the basic section B, an MPEG signal comprising a video signal V and motion 
vectors MV, is applied to a decoder DEC in order to retrieve the video signal V in 
dependence upon the motion vectors MV from the MPEG input signal. The video signal V is 
applied to a video post-processor VPP to obtain an output video signal OVS having a 

25 doubled field frequency and/or line frequency or an improved motion portrayal with respect 
to the video signal V. Alternatively or in addition, the video post-processor VPP reduces 
noise and/or changes the number of pixels per line and/or lines per field of the video signal 
V. The video post-processing carried out by the processor VPP is dependent on improved 
motion vectors MV\ The output video signal OVS is displayed on a display device D. 

30 In accordance with a first embodiment B-hEl, the improved motion vectors 

MV* arc obtained from the motion vectors MV contained in the MPEG signal by means of a 
post-processor PP. For example, a snrwothing operation will reduce most of the artifacts 
which would arise if the MPEG motion vectors MV were directly used for the video post- 
processing carried out by the processor VPP. A smoothing operation may include separate 
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low-pass filtering operations on the horizontal and vertical components of the motion vectors. 
Alternatively, the horizontal and vertical components of the'motion Vectors can be applied to 
median filten to ttphct extreme motion vector component values by existing motion vector 
component values of motion vectors of neighboring blocks of picture elements. 

5 In accordance with a second embodiment B+E2 which is covered by a 

copending patent appUcaticMi (attorneys' docket PHN 16,106). the MPEG motion vectors MV 
arc multiplexed by a multiplexer MUX with candidate vectors generated by a candidate 
vector generator CVG, optionally after a (e.g. smoothing) post-processing operation carried 
out by the post-processor PP. The mult^)lexed vectors arc applied to a candidate vector 

10 selector CVS to obtain improved motion vectors MV' which are expected to show the 
smallest errors. 

In a third embodiment B+E3 which is covered by the claims of the present 
patent plication, a parameter extractor PE extracts motion parameters pi pn from the 
MPEG motion vectors MV. The motion parameters pi .. pn are applied to a vector field 

15 calculator VFC to obtain the improved motion vectors MV\ The predetermined motion 
vector recited in claim 1 may be the zero vector. 

In a fourth embodiment B+E3+E4 which is covered by the claims of the 
present patent a^yplicat^on* ^ motion vectors calculated by the vector field calculator in 
section E3 arc multiplexed with candidate motion vectors from the candidate vector generator 

20 CVG, and applied to the candidate vector selector CVS to obtain the improved motion 
vectors MV\ This fourth embodiment combines the features of the second and third 
embodiments. 

The basic considerations underlying the present aspect of die invention can be 
summarized as follows. Different motion estimation algorithms will have different 

25 applications based on the properties of their resulting vector fields. MPEG video daU streams 
contain motion vectors, which can possibly be used in high quaUty video processing. 
However, it is likely that these MPEG motion vectors are optimized to achieve an efficient 
compression, so that these vectors cannot be expected to represent the true motion. This 
means that additional processing is needed to improve their quality and usability for a high 

30 quality post-processing. It is proposed to process the MPEG motion vectors at the receiver 
side. More specifically, the MPEG vectors are used to generate motion parameters which are 
used to obtain one or more parametric motion vectors which can be used as additional 
candidate vectors in an efficient motion estimator. In this manner, the applicability of MPEG 
motion vectors is extended. Obviously, the present aspect of the invention is not limited to 
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MPEG signals; other signals can be treated in the same manner. Reference is made to 
copKiding patent applicadons for features disclosed but not claimed in this application. 

VT. rnnclusion and final remarks 
5 A new motion estimator has been presented. This estimator selects its motion 

vector for the current image from four candidate vectors only, and is characterized in that 
OTc of these candidate vectors is produced with a parametric model describing the global 
(camera) motion in the previous image. 

We introduced this "parametric candidate* in a very efficient (3-D Recursive 
10 Search) block-matching algorithm. To this end, we block-altematingly rq>laced one of the 
candidates of this estimator by a candidate generated with a four parameter model. The 
parameters for the model were extracted once per field using nine widely spaced sample 
vectors only, taken from the previously estimated vector field. This results in an almost 
negligible additional processing power requirement for the eventual algorithm. We showed 
15 that, with these 9 extracted motion vectors, it is possible to generate 18 sets of 4 parameters 
describing the camera motion. A median operation was introduced to select the applied 
parameter set, thus eliminating the outliers due to object motion. Furthermore, we showed 
that knowledge of the horizontal and vertical sampling densities could be used to judge the 
reliability of the model. This information provided a means to switch-off the block- 
20 alternating injection of the "parametric candidate" in the estimator in case it could lead to a 
degradation of the estimator performance. 

In the evaluation part IV, a significant advantage, up to 50% reduction in MSB, 
was found on critical material applying the motion vectors for de-interlacing. A photograph 
of the vector field enabled a subjective evaluation, suggesting that also the fractional part of 
25 the displacement vectors obtained from the resulting block-matcher closely correspond to the 
true-motion in the sequence. Finally, it was shown tiiat the performance of the reliability 
indicator, although working properly in the experiments, was not too critical. 

A preferred embodiment of the invention, providing an efficient true-motion 
estimator using candidate vectors from a parametric motion model, can be summarized as 
30 follows. Efficient motion estimation algorithms are known that select their output motion 
vector from a limited number of likely correct candidate, or prediction, vectors. In addition 
to the known spatial and temporal prediction vectors, an additional and independent 
prediction is proposed. This candidate is generated witii a parametric model describing the 
global motion in a previously estimated motion field. The proposal is elaborated as an 
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addition to the 3-D Recursive Search block-matching algorithm. The evalyation shows that a 
sub-pixel accurate, true-motion estimator results with a very low operations cotint. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention, and that those skilled in the art will be able to design many alternative 
5 embodiments without dqarting ftom the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be construed as limiting the claim. The 
invention can be implemented by means of hardware comprising several distinct elements, 
and by means of a suitably programmed computer. 

In the present disclosure, it has been described to use parameters determined on 
10 the basis of motion vectors obtained for a preceding field n-1, to obtain motion parameters 
used in determining an additional vector for the present field n. However, in a conversion 
from a 50 Hz input signal origiruting from a 24 Hz movie film, to a 100 Hz output signal, 
motion vectors determined during a first estimation stage for a given field may be used to 
obtain parameters for use in a second estimation stage for the same field. Thus, it may 
15 h^pen that field n-1 is the same field as field n. 

While a preferred application of the present invention is in a recursive motion 
estimator, the additional motion parameter-dq)endcnt vector may also be ^plicd in other 
motion estimators like the well-known 3-step block matching algorithm. Thus, the expression 
predetermined motion vector in the claims includes candidate motion vectors obtained in 
20 accordance with any algorithm other than an algorithm in which motion vectors arc derived 
from motion parameters. 

In another embodiment in accordance with the present invention, the motion 
parameters are obtained in accordance with a 2-D histogram algorithm to which all motion 
vectors for the given field (n-1) are ^plied, the peaks in the histogram forming the motion 
25 parameters. The peaks indicate motion vectors which very often occur in the image and 
which probably describe a global motion. The motion vectors indicated by the peaks, and/or 
points of gravity of a cluster of peaks, can be used as additional parametric motion vectors. 
If there are more peaks than additional nwtion vectors required, the motion vectors indicated 
by the peaks can be used in a cyclic manner as additional motion vectors. 
30 The motion vector estimation in accordance with the present invention is 

preferably applied in a scan format conversion for e.g. converting a 50 Hz field frequency 
signal into a 100 Hz field frequency signal, for converting a 60 Hz field frequency signal 
into a 50 Hz field frequency signal, or for converting an interlaced signal into a 
progressively scarmed signal. 
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Claims: 

' I 

1. A method of estimating motion vectors, comprising the steps of: 
detennining (PC) motion parameters (pl-p4) for a given field (n-1) of a video 

signal; and 

determining motion vectors ( n) ) for a field (n) of said video signal based 
5 up<m at least one predetermined motion vector ( D(x-X, y-Y, n), Ii(x+X, y-Y, n), 
n(x, y+2Y, n-1) ), and at least one additional motion vector ( PVfX) ) derived (PG) from 
said motion parameters (pl-p4). 

2. A method as claimed in claim 1, wherein said at least one predetermined 
10 motion vector ( D(x-X, y-Y. n). E(x-»-X, y-Y. n). I2(x. y+2Y, n-1) ) includes at least a 

first previously determined motion vector ( I2(x. y-l-2Y. n-1) ) corresponding to a selected 
block of pixels out of a plurality of spatio-temporally neighboring blocks, and a second 
motion vector obtained by adding (UP-MUX) an update vector ( IKX, n) ) to the first or 
another previously determined motion vector ( Ii(x-X, y-Y, n), D(x+X, y-Y. n) ). 

15 

3. A method as claimed in claim I. wherein said motion vectors for a subsequent 
field (n) of said video signal are based (SEL) upon a selection from said at least one 
predetermined motion vector ( D(x-X, y-Y, n), D(x-I-X, y-Y, n), E(x, y-H2Y, n-1) ), and 
said at least one additional motion vector ( EYGO ) derived (PG) from said motion 

20 parameters (pl-p4), 

4. A method as claimed in claim 1, wherein said motion parameters (pl-p4) for 
said given field (n-l) of a video signal are determined (PC) from motion vectors determined 
for said given field (n-l). 



25 



5. A method as claimed in claim 4, wherein said motion parameters (pl-p4) are 

calculated (PC) by detennining for each motion parameter, a plurality of parameter 
candidates from selected motion vectors, and by taking a median of said pluraUty of 
parameter candidates to obtain the corresponding motion parameter. 



30 
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6. A method as claimed in claim 4, wherein said motion parameters ^l-p4) are 

calculated (PC) by the steps of: 

calculating a plurality of horizontal and vertical difference vector components 
from a plurality of selected motion vectors in the given field; 

selecting horizontal and vertical vector components corresponding to minimum 
difference vector componenu to obtain selected horizontal and vertical vector components; 
and 

calculating said motion parameters in dependence upon said selected horizontal 
and vertical vector components. 



10 



7. A method as claimed in claim 4, wherein said motion parameters (pl-p4) arc 
calculated (PC) by determining for each motion parameter, a plurality of parameter 
candidates from selected motion vectors, and by cyclically applying said pluraUty of 
parameter candidates to obtain the corresponding motion parameter until the parameters 

15 correspond to an aspect ratio of said video signal. 

8. A method as claimed in claim 4, wherein said motion paiamcters result from a 
two-dimensional histogram operation on said motion vectors determined for said given field 
(n-1). 



20 



9. A method as claimed in claim I, wherein said motion parameters (pi .. pn) are 

determined from motion vectors (MV) received with said video signal (MPEG). 



10. A motion vector estimation device (ME), comprising: 

25 means (PC) for determining motion parameters (pl-p4) for a given field (n-1) of 

a video signal; and 

means (SEL) for determining motion vectors ( D(X, n) ) for a field (n) of said 
video signal based upon at least one predetermined motion vector ( E(x-X, y-Y, n), 
Ii(x+X. y-Y, n), I2(x, y+2Y, n-1) ), and at least one additional motion ( Eit(X) ) vector 
30 derived (PG) from said motion parameters (pl-p4). 
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11. A video display apparatus, comprising: 

. an input (A) for receiving a video signal at a first scan format; 
a motion-compensated interpolation device (ME, FM2, I) coupled to said input 
for providing an output video signal at a second scan format different from said first scan 
5 format; and 

a display (Dp) for displaying said output video signal at said second scan 
format; characterized in that 

said motion-compensated interpolation device (ME, FM2, I) includes a motion 
vector estimation device (ME) as claimed in claim 10. 
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